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Abstract 


The Federal Pell Grant Program is the nation’s largest need-based grant program. While 
students’ initial eligibility for the Pell is based on financial need, renewal of the award is 
contingent on their making satisfactory academic progress (SAP) — meeting minimum academic 
standards similar to those proposed in models of performance-based scholarships. It is not clear 
how many students are affected by failure to meet SAP standards, or how the policies shape 
student outcomes. In this study, we draw from literature on performance-based funding and 
academic probation to consider the potential implications of SAP standards. We describe federal 
guidelines and illustrate how SAP is evaluated in a statewide community college system. Using 
administrative data with term-by-term measures of Pell receipt, student grades, attempted and 
earned credits, persistence, degree attainment, and transfer, we employ regression discontinuity 
and difference-in-differences approaches to examine the magnitude of SAP failure and its 
effects. Our results suggest that a substantial portion of Pell recipients at community colleges are 
at risk for Pell ineligibility due to their failure to meet SAP grade point average (GPA) or credit 
completion requirements. Approximately a quarter fail to meet the GPA standard alone. When 
the credit completion requirement is taken into consideration, the first-year SAP failure rate 
approaches 40 percent. Our preferred difference-in-differences estimates show mixed effects of 
SAP standards: Failing to meet the GPA requirement has a negative impact on persistence into 
the second year, but it may improve associate degree attainment and transfer among students 
who are not discouraged from reenrolling. 



Table of Contents 


1. Introduction 1 

2. Background 3 

Students From Low-Income Families, Need-Based Aid, and Navigating College 3 

Academic Performance Standards: Incentivizing Effort and Discouraging Persistence? 4 

Satisfactory Academic Progress, as Defined by the Pell Grant Program 7 

3. Data 9 

4. Methods 12 

Regression Discontinuity 13 

Difference-in-Differences 1 5 

5. Results 17 

How Many Students Are Potentially Affected by SAP Requirements? 17 

What Happens to Students Who Fail SAP Over Time? 19 

6. Discussion 25 

References 28 

Appendix A: Regression Discontinuity Results 33 

Appendix B: Difference-in-Differences Results 38 




1. Introduction 


Many students start college but leave without degrees (Attewell, Heil, & Reisel, 2011; 
Deil-Amen & DeLuca, 2010). College persistence rates are especially low among students who 
stand to gain the most from postsecondary education: those from low-income families (Bailey & 
Dynarski, 2011; Brand & Xie, 2010). While some researchers are skeptical of the relationship 
between financial constraints and educational outcomes (Cameron & Heckman, 1998; Carneiro 
& Heckman, 2002; Heckman, 2000; Mayer, 2010), rapidly shifting financial demands, including 
rising tuition rates, mean that financial constraints are more likely to impact students than ever 
before. Research consistently suggests that low-income families are particularly sensitive to 
college prices, even after initial college enrollment (Bowen, Chingos, & McPherson, 2009; 
Deming & Dynarski, 2010). Need-based financial aid is a widely used tool to promote college 
attainment, with the federal government delivering $185.1 billion in aid to undergraduates in 
2013 (Baum & Payea, 2013). 

The Federal Pell Grant Program is the single largest source of need-based financial aid 
for college students in the United States, providing over $32 billion in grants to nearly 9 million 
undergraduates (over a third of undergraduates in the 2012-2013 academic year) (Baum & 

Payea, 2013). While eligibility is initially based purely on financial need, recipients must meet 
satisfactory academic progress (SAP) requirements in order to remain eligible for Pell Grants 
beyond the first year. Institutions have flexibility regarding how they define SAP but commonly 
require students to maintain a cumulative grade point average (GPA) of 2.0 or higher and to 
complete at least two thirds of the course credits that they attempt. Those who fail to meet the 
SAP requirements at the end of the institution’s evaluation period may continue to receive aid for 
one additional term, but if they are still failing SAP at the end of that tenn, they may lose 
eligibility for Pell. 

The academic requirements for Pell Grant renewal have received little attention from 
researchers and policymakers, despite their potentially broad consequences. Forty-five percent of 
Pell Grant recipients entering postsecondary institutions do not enroll for a second year of 
college (authors’ calculations). 1 While perfonnance-based scholarships have risen in popularity, 
the perfonnance-based aspects of the nation’s largest need-based grant program have gone 
virtually unstudied. Do the bureaucratic hurdles posed by SAP standards minimize the PelFs 
effectiveness? Do they contribute to the observed low rates of persistence? The prevalence of 
SAP failure and its consequences in terms of Pell loss, persistence, subsequent aid receipt, and 
degree attainment are unknown. 

To our knowledge, this is the first empirical examination of SAP policy since the 1980s 
(when, soon after the onset of the federal policy, a few institutions examined the effects of SAP 

1 The average rate of reenrollment among Pell recipients was calculated using data from the 2004/2009 
Beginning Postsecondary Students Longitudinal Study. The estimates for students attending public two-year 
colleges, which enroll a disproportionate share of financially disadvantaged students, were similar — approximately 
46 percent of Pell recipients at community colleges do not enroll for a second year. 
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implementation) and the first to use multi-institutional data. Since SAP policy was first enacted, 
enrollment in college has soared. Educational expansion and increasing diversity among college 
students, twinned with rising college costs, mean that more students than ever face the potential 
repercussions of SAP policy. In light of these trends, there is a clear need for more information 
about SAP policy and its effects on student outcomes; this paper makes several contributions. 
First, we clearly outline the federal guidelines for SAP evaluation by postsecondary institutions 
receiving federal financial aid and describe how evaluation is implemented in a state community 
college system. Second, we theorize how SAP policy may impact student outcomes, drawing 
from prior research on performance-based aid and academic probation. Third, we provide 
evidence regarding the prevalence and consequences of SAP failure, using both national data and 
state administrative data from community colleges. 

We find that approximately one in five (21 percent) first-year Pell recipients nationally 
are at risk of losing their Pell due to the SAP GPA criterion alone. The rates of first-year SAP 
failure are increasing over time, particularly at community colleges, which in recent years have 
had the most students at risk (although SAP failure is not unique to these institutions). At 
community colleges, approximately 25 percent of first-year Pell recipients fail to attain a 2.0 
GPA in both national and state administrative data. When we incorporate the credit completion 
requirement, the overall first-year SAP failure rate approaches 40 percent in our administrative 
data. It is worth noting that poor performance is by no means limited to Pell recipients; the rates 
of low academic achievement (i.e., falling below a 2.0 cumulative GPA) are only modestly lower 
among first-year students not receiving the Pell. 

To estimate the consequences of SAP failure, we use both regression-discontinuity (RD) 
and difference-in-differences (DD) approaches. The RD approach compares outcomes for Pell 
recipients just above and below the SAP GPA criterion, while the DD approach compares these 
differences against those for non-Pell recipients (who are also subject to academic probation but 
do not face the same loss of financial aid). The pattern of results from the two approaches is 
somewhat different; however, the RD estimates are much less precise (and not statistically 
significant). Our preferred DD estimates suggest that failing SAP (and thus facing the loss of 
Pell, beyond any general effects of academic probation) may have heterogeneous effects, 
decreasing persistence into the subsequent year but increasing eventual degree attainment for 
those who remain. 

The patterns we describe require careful interpretation. Failing SAP does not necessarily 
imply that a student’s decision to enroll in college was a mistake. College is necessarily an 
experiment; much like investments in small business, some failures are to be expected, and 
failure does not mean the investment was flawed (Manski, 1989). Even students with low grades 
and minimal credits may receive some payoff in the labor market relative to students who never 
attended college. Moreover, while failing SAP at the end of the first year may have negative 
consequences for some students, it is possible that the existence of performance standards shifts 
the entire distribution of performance up in the first year, so that fewer students are near the 
cutoff in the first place (an effect which is outside the scope of the present analysis). At the same 
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time, it is important to note that, while poor academic performance is widespread (across 
institutional sectors and student demographics), SAP policy targets undergraduates from 
America’s most disadvantaged families (most Pell recipients come from families earning under 
$40,000 annually) — those in greatest need of support in order to attain a degree and who stand to 
benefit the most from degree attainment (Baum & Payea, 2013). The main conclusions we can 
draw at this point are simple: Many Pell recipients are at risk of losing aid due to SAP failure, 
and this appears to have consequences for students’ future persistence and degree attainment. It 
is not clear that policymakers, college administrators, and students are aware of the magnitude of 
this risk. 


2. Background 

Students From Low-Income Families, Need-Based Aid, and Navigating College 

Despite national efforts to improve college attainment, including significant spending on 
financial aid over the last 40 years, the relationship between family income and college 
attainment is stronger than ever (Alon, 2009; Bastedo & Jaquette, 2011). While the enrollment 
gap between children bom in the bottom and top income quartiles appears to be shrinking (albeit 
slowly), the completion gap is large and persistent. This is likely because students from low- 
income families face considerable hurdles to persisting once enrolled in college. 

Many low-income students are first-generation college attendees and may be less 
psychologically and culturally prepared to navigate college (Haveman & Smeeding, 2006). 
Students from low-income families may arrive at college academically underprepared for 
postsecondary coursework, due at least partially to socioeconomic residential segregation and 
their concentration in resource-constrained school districts (Goldrick-Rab, Kelchen, Harris, & 
Benson, 2014; Lareau & Cox, 2011; Long, Conger, & Iatarola, 2012). Low-income students are 
also more likely to work for pay during college than their peers, minimizing the time they have 
available to study and to interact with faculty and classmates (Belley & Lochner, 2007; Benson 
& Goldrick-Rab, 2011; Bozick, 2007; Roksa & Velez, 2010). 

Need-based financial aid aims to offset the challenges faced by low-income students by 
alleviating financial constraints. This public policy is one of the most expensive higher education 
policies in the United States, but mounting evidence suggests it has the potential to reduce 
socioeconomic gaps in college outcomes. Financial aid through grants is linked with reduced 
college dropout behavior (Bettinger, 2004); decreased need to work for pay (Benson & Goldrick- 
Rab, 2011; Schudde, 2013); and increased attendance, persistence, credit accumulation, learning, 
and degree attainment (Arum & Roksa, 2011; Castleman & Long, 2013; Dynarski, 2003; 
Goldrick-Rab et ah, 2014). This may be particularly true at community colleges. Despite 
arguments that “community college tuition is cheap” and that financial aid is “plentiful,” 
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additional aid dollars appear to be the largest detenninant of students’ graduation prospects, 
according to recent research by Attewell et al. (2011, p. 550). 

Much of our knowledge about need-based grants focuses solely on the impact of 
providing money, as though it were offered with no strings attached. But recent evidence 
suggests that the “strings” — in this case, the bureaucratic hurdles for award renewal — are 
potentially quite important. Today’s financial aid system is complex for both students and 
administrators to navigate (Dynarski & Scott-Clayton, 2006, 2008; Goldrick-Rab, 2013a; 
Goldrick-Rab et al., 2014). For students, the uncertainty that comes with financial aid, in which 
awards have different eligibility standards, require reapplication for renewal, and may vary in 
amount over time, can create a “trajectory of college enrollment fraught with uncertainty, stress, 
and even fear” (Goldrick-Rab & Kendall, 2014, p. 1 1). While some research, reviewed below, 
suggests that linking financial incentives to academic requirements positively influences student 
outcomes, the complex requirements of financial grants may also have negative impacts. 

Much of the scholarship highlighting the complexity in the Pell Grant program focuses 
on the process of applying for financial aid and detennining eligibility status, and the subsequent 
impacts on enrollment for low-income students (e.g., Bettinger, Long, Oreopoulos, & 
Sanbonmatsu, 2012; Dynarski & Scott-Clayton, 2006; Dynarski & Wiederspan, 2012). But there 
are also complexities in the process for retaining financial aid, as many awards have different 
academic requirements (Goldrick-Rab, 2013b; Goldrick-Rab et al., 2014). Competing award 
requirements may be difficult to master and interpret. Because the purpose of financial aid is to 
offset financial constraints for college students, federal financial aid policy should, at the least, 
be designed to do no harm to students (Scott-Clayton, 2013). But complexity of navigating 
financial aid, even after initially receiving the award, may undermine this simple goal. In the 
next section, we theorize how tying academic performance to need-based financial aid influences 
student outcomes by examining two competing, though not mutually exclusive, hypotheses. 

Academic Performance Standards: Incentivizing Effort and Discouraging 
Persistence? 

SAP standards complicate the already intricate process of using federal financial 
assistance to attend college. Extant research on SAP policies is sparse, dated, and limited to 
descriptive studies of policy implementation within individual institutions (e.g., Bennett & 
Grothe, 1982; McNair & Taylor, 1988). To guide our thinking as we examine the potential 
implications of SAP policies, we turn to related literature about performance standards for 
financial aid recipients and academic probation policies. 

Performance-based aid. Scott-Clayton (2011a) argues that financial aid with academic 
requirements may improve student outcomes through two potential mechanisms. First, financial 
aid offsets the cost of college, reducing the need for student employment. This enables students 
to spend more time on coursework, potentially speeding their progress toward a degree. Second, 
annual achievement requirements for award renewal may offer direct incentives to increase 
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academic effort, aligning students’ incentives with their funders’ preferences (Scott-Clayton, 
2011a, p. 615). 

Research on West Virginia’s PROMISE scholarship suggests that both mechanisms, cost 
reduction and incentivized effort, may be at play (Scott-Clayton, 2011a). While the PROMISE 
scholarship requires that students qualify for the program by meeting initial academic 
requirements, it also specifies that they must maintain a minimum GPA and course load to retain 
the scholarship. In her evaluation of PROMISE’S effects, Scott-Clayton (2011a) suggests that the 
program’s substantial impacts on cumulative GPA and total credits earned in the first year, along 
with its moderate four-year impacts on GPA and credit accumulation, support the notion that 
academic incentives promote improved academic achievement. The null impacts of PROMISE 
receipt on persistence and weekly school-year earnings — the two measures that are likely most 
sensitive to cost-of-college effects — suggest that reducing cost of attendance alone may not 
induce improvements in college outcomes, though, as Scott-Clayton notes, “this hardly implies 
that costs are irrelevant. Perhaps the only reason why incentives work is because students face 
credit constraints” (p. 644). 

Still, similar research on Georgia’s HOPE Scholarship Program finds that most merit aid 
recipients lose their scholarships due to eligibility requirements for renewal. Henry, Rubenstein, 
and Bugler (2004) argue that the positive impacts of the scholarship on persistence and 
graduation are only significant for those who maintain eligibility. In other words, retention of 
financial aid is critical to improving the outcomes of students. 

It is difficult to generalize the findings of the PROMISE or HOPE programs to recipients 
of need-based aid because students receiving need-based financial aid differ systematically from 
those who are not receiving need-based aid. The background characteristics of need-based aid 
recipients are highly correlated with dropout behavior (Bettinger, 2004). While PROMISE-style 
programs illustrate the effectiveness of incentives for a population with demonstrated prior 
academic success, we cannot necessarily apply the same expectations to other populations. 
Furthermore, the success of many merit-based and performance-based scholarships “relies in part 
on the existence of a wholly need-based program like Pell that serves as the foundation of 
financial support” (Baum & Scott-Clayton, 2013, p. 1 1; emphasis added). As we show below, 
however, the Pell program is performance-based when it comes to renewal beyond the first year. 

Perfonnance-based scholarships have some striking similarities to the Pell program. 
Unlike merit scholarships, they are not awarded based on prior achievement; rather, they are 
need-based grants that students can earn by meeting specific academic benchmarks over the 
course of a tenn (Welbeck, Ware, Cerna, & Valenzuela, 2014). In MDRC’s recent guide for 
establishing performance-based scholarship programs, the authors mention enrollment in a 
minimum number of credits and course grades (such as a GPA of at least 2.0) as frequently used 
benchmarks (Welbeck et ah, 2014, p. 2). The benchmarks are quite similar to those already 
imposed upon Pell recipients. 
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Proponents of performance-based scholarships argue that the awards fill the gap between 
students’ level of need and their level of financial aid (as they, unlike the Pell, are a form of 
supplemental aid). They offer a “meaningful incentive to stay in school and make progress 
toward a degree” by encouraging students to focus on their studies (Welbeck et ah, 2014, p. 4). 
Financial incentives are offered at multiple time points throughout the semester if students meet 
performance benchmarks (Patel, Richburg-Hayes, de la Campa, & Rudd, 2013). Advocates 
expect to see better performance in courses; faster progression through degree requirements; and, 
in the long term, higher rates of graduation and transfer to four-year colleges (Patel et ah, 2013). 

Performance-based scholarship programs appear to meet some of those expectations. The 
Opening Doors Demonstration in Louisiana increased rates of retention in the second semester 
and credit accumulation in the first two semesters (Richburg-Hayes et ah, 2009). The promising 
short-term results inspired the launch of the national Performance-Based Scholarship 
Demonstration in 2008, which aimed to evaluate the effectiveness of perfonnance-based 
scholarships among low-income students in various geographical locations with different 
amounts of money over different durations of time. The Performance-Based Scholarship 
Demonstration results, thus far, are modest. While the scholarships increased credits earned in 
the first year, they do not appear to have significantly increased persistence into the second year 
of college (Patel et ah, 2013; Patel & Valenzuela, 2013). 

Academic probation. Perfonnance-based scholarship advocates argue that performance 
standards should incentivize effort through the positive rewards they offer for those who meet 
performance standards. But what happens when policies are structured to provide negative 
consequences for failure to meet standards? Academic probation policies often follow that 
structure — students with a GPA below a certain threshold are placed on probation and met with a 
threat of punishment (for instance, suspension) if they fail to improve their grades. The driving 
theory behind performance standards such as those employed in academic probation is that they 
weed out students who have little chance of success and motivate those most likely to succeed 
(Benabou & Tirole, 2002). It is likely that the motivation for the SAP requirements for Title IV 
recipients is similar, though we cannot ascertain the motivation from policy documents. 

Recent research suggests that when academic standing is evaluated at the end of the first 
year, it discourages some students from returning while improving the performance of others 
(Lindo, Sanders, & Oreopoulos, 2010). The authors evaluate the academic probation policy at 
three campuses of a large Canadian university, using a regression discontinuity research design. 
Comparing students with a GPA just above the academic probation threshold with those whose 
GPA is just below the threshold, they find that some students are discouraged from returning, but 
those who reenroll improve their academic performance. At first glance, this aligns with the 
“weeding out” hypothesis. However, the impacts of academic probation vary across students 
based on prior achievement, gender, and native language. Of greatest note, academic probation 
appears to have a strong negative effect on degree attainment among high-ability students (i.e., 
students with higher achievement in high school). Overall, the authors conclude that being 
placed on academic probation negatively impacts students’ perceptions of their ability, and that 
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students with prior academic success were subject to greater shock when notified of their failure 
to meet academic standards (Lindo et ah, 2010, p. 23). 

SAP standards have many similarities to academic probation standards. Much like being 
put on academic probation, failing to make satisfactory academic progress has delayed 
consequences. Initially, the only consequence is notification of failure to meet academic 
expectations. Notification alone may discourage some students from returning and have the 
unintended consequence of discouraging students who could otherwise succeed in college. The 
most severe consequence of failing to meet SAP standards, ineligibility for federal aid, is further 
delayed. If students become ineligible for the Pell Grant, their net price of attending college 
increases by at least the size of their Pell Grant (or more, if they received other sources of 
financial aid that were tied to Pell receipt). 

While the threat of Pell loss could serve as a motivation to improve student effort and 
academic progress, this is more likely occur if aid recipients who fail to meet academic standards 
early in college are embedded in strong academic support and communication systems (i.e., if 
students receive notification of their status and what it means, and are connected with the 
academic supports necessary to improve their status). Institutional policies may recommend 
academic intervention for students who fail to meet SAP standards and appeal for financial aid 
renewal. However, offices on community college campuses are often “loosely coupled” — the 
actions of one office may not reflect those in another (Meyer & Rowan, 1978; Weick, 1976). If 
the financial aid officers and the academic advisers locally adapt policies with little 
communication between the offices, early academic intervention is unlikely (Goldrick-Rab, 
2013b). A loosely coupled system may contribute to information constraints where students are 
not entirely aware of institutional SAP policies, how SAP status affects Pell eligibility, and how to 
improve their SAP status; subsequently, students in loosely coupled systems struggle to navigate 
college pathways (Rosenbaum, Deil-Amen, & Person, 2006; Scott-Clayton, 2011b, 2013). 

Under current institutional policy in our state sample (details of which are explained in 
the next section), students who fail to meet SAP requirements after their first semester receive a 
written warning, followed by a semester-long “warning period.” Those who continue to fail to 
meet standards risk losing Pell eligibility the following semester. There are no fonnal academic 
interventions for Pell recipients who fail to meet SAP requirements. 

Satisfactory Academic Progress, as Defined by the Pell Grant Program 

Origins of SAP and federal/institutional roles. In 1976, the tenn “academic progress” 
was introduced into federal financial aid legislation as an amendment to the Higher Education 
Act of 1965. The amendment stipulated that, to receive financial assistance under Title IV, 
students must demonstrate “satisfactory progress” toward a degree (Bennett & Grothe, 1982). 

The legislation did not explicitly define satisfactory progress, but the Interim Final Regulations 
in 1978 stated that an institution should evaluate progress by establishing a “normal time frame” 
for completing the course of study and by measuring grades or work “against a norm” (Bennett 
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& Grothe, 1982, p. 1). Since then, the policy has evolved to be more explicit in its terms while 
still allowing institutions to determine, within certain confines, how best to evaluate academic 
progress. 

Today, students receiving federal financial assistance must be evaluated at the end of 
each academic year by their institution. Per the Higher Education Act, an institution’s definition 
of SAP must, at the least, align with its standards for graduation, although the timeline for 
evaluation differs (Student Eligibility, 2013, p. 571). The Code of Federal Regulations provides 
some additional guidance for institutional evaluation of student progress. Institutions must set a 
“qualitative standard,” in which students must meet a GPA of 2.0 or its academic equivalent by 
the end of their second academic year; a “quantitative standard,” in which there is a “minimum 
percentage of work” successfully completed; and a “maximum timeframe” standard, which 
requires students to finish a program within at least 150 percent of its published length 
(Satisfactory Academic Progress, 2013, p. 546). 

The language of the federal policy offers some flexibility. SAP must be evaluated at the 
end of each academic year, but the detennination of satisfactory progress is left up to the 
institution, so long as it requires a 2.0 cumulative GPA by the end of the second year, assesses 
how much work students complete, and sets a maximum time frame standard. Perhaps more 
importantly, the institution’s policy must be “at least as strict” as the academic policy applied to 
students not receiving assistance from Title IV programs (Satisfactory Academic Progress, 2013, 
p. 546). 

Students who do not meet an institution’s SAP standards become ineligible to receive 
federal financial aid. However, students able to demonstrate they faced “undue hardship” may be 
exempt from the consequences, at least initially (Student Eligibility, 2013, p. 571). The federal 
legislation includes an appeals clause, allowing institutions to waive the provisions of the SAP 
policy if a student faces the death of a relative, personal injury or illness, or “any special 
circumstances as detennined by the institution” (Student Eligibility, 2013, p. 571). 

In 201 1, the federal policy was changed to include a warning period. The new legislation 
requires institutions to warn students prior to the termination of federal funding. While the policy 
change does not, technically, require more frequent assessment of SAP, it may in practice. 
Evaluation now “must occur at the end of each payment period” for programs longer than an 
academic year (Rules and Regulations, 2010, p. 66882). Institutions evaluating SAP at the end of 
the academic year may need to evaluate more frequently in order to offer students a warning 
period before financial aid termination (Rules and Regulations, 2010). Many critics responded 
that the “one-semester rule” had a “profound” impact on their students (Ashford, 2012). Under 
the previous policy, some colleges elected to evaluate students at the end of each academic year, 
and then applied a warning period to those who failed to meet SAP after that point. For those 
colleges, the new policy shifts the warning period one semester earlier. 

Rules in effect for students in our state administrative sample. In this study, we 
explore the application of SAP guidelines within a state community college system that evaluates 



satisfactory academic progress at the end of each term. Per the policy, students are meeting SAP 
requirements at the end of each term if they have a cumulative GPA of 2.0 or higher, complete at 
least 67 percent of all credit hours attempted, and complete their program of study within 150 
percent of the expected time frame. At the end of each tenn, the college notifies students of their 
SAP status. The statuses include satisfactory, warning, termination, and probation-by-appeal . 
Students who meet the minimum SAP standards are deemed satisfactory, while students who 
have not earned the required GPA or credit completion rate are placed on warning for the 
subsequent semester. Students on warning are able to receive aid but risk financial aid 
tennination if they continue to fail to meet SAP standards. Upon failing to meet SAP standards 
during their warning term, they must appeal to continue receiving federal and state financial aid. 
Financial aid is terminated for students who do not appeal or who make an unsuccessful appeal 
attempt. Students with successful appeals enter a probation-by-appeal period of one tenn. During 
this period, they receive financial aid but must create an academic plan to meet SAP 
requirements. There are no set policies requiring a specific academic intervention for students on 
probation, just the “academic plan” worked out with a financial aid officer. Students on 
probation-by-appeal who fail to meet SAP standards lose financial aid eligibility. 


3. Data 

We use several sources of data to explore the prevalence and consequences of SAP 
failure. To highlight national trends, we rely upon three waves of data on undergraduates from 
the National Postsecondary Student Aid Survey (NPSAS) in 2004, 2008, and 2012. The NPSAS 
examines the characteristics of a nationally representative sample of postsecondary students. It 
includes data on students’ background, academics, and financial aid for a single year and collects 
information from a range of sources, including student surveys, academic transcripts, and 
administrative financial aid records. 

While the NPSAS is nationally representative of college students, it includes older 
students who have prior college experience. Because many students drop out of college after the 
first year, the NPSAS sample overall is more academically selective (because it includes second 
years, third years, etc.) than a sample of beginning students, such as the Beginning 
Postsecondary Students Longitudinal Study. In our analysis of national trends, we limit the 
NPSAS sample to “first-year” students; however, the variable we use in the NPSAS data is based 
on academic standing rather than timing of initial enrollment. As a result, the NPSAS “first-year” 
students are not precisely equivalent to students who are in their first academic year. 

To explore the prevalence, trends, and consequences of SAP failure at a more granular 
level, we turn to administrative data from a state community college system (SCCS) that 
includes over 50 separate public two-year institutions. We restrict the sample to focus on cohorts 
with at least three years of follow-up data to track enrollment and transfer behavior, as well as 
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early degree attainment. The analytic dataset includes a pooled sample of 173,312 students, 
50,008 of whom received Pell Grants, from six cohorts of fall-term enrollees (2002-2007). 

The state administrative data track federal and state aid and transcript measures, 
including courses, credits attempted and completed, and grades for each term. The data are 
derived from three sources: SCCS data with student information on demographics, transcripts, 
and financial aid; National Student Clearinghouse graduation data; and Employment Security 
Commission earnings data. Our analytic dataset includes student background measures, 
including race (collected by the SCCS as White, Black, “other race,” or Hispanic), gender, age at 
college entry, state residency status, dependency status for financial aid purposes, and high 
school graduation status. We are able to capture whether students work for pay while enrolled, 
their earnings during their first semester (as a proxy for the amount worked during initial 
enrollment), and their expected family contribution (EFC) for financial aid purposes. Finally, we 
track students for three years after college entry to paint a picture of how SAP failure impacts 
college outcomes. To understand how SAP failure impacts immediate persistence and 
achievement, we examine enrollment and GPA in the first term of the second year of college. To 
explore the implications of SAP policy for early degree and transfer outcomes, we examine 
certificate and associate degree attainment and transfer to a four-year college. 

To demonstrate how the students in our state administrative data compare to community 
college students across the nation, we present sample means for community college students 
(broken into Pell and non-Pell students, based on Pell receipt at college entrance) in the 
2004/2009 Beginning Postsecondary Students Longitudinal Study (BPS:04/09) and our pooled 
state administrative data (see Table 1). The BPS is a longitudinal study following a nationally 
representative sample of first-time college students, a subset of students from the NPSAS 2004. 

It includes data on students’ background, academics, and financial aid, drawing information from 
a range of sources, including student surveys, academic transcripts, and administrative financial 
aid records. 

Students in the SCCS sample are, on average, older than those in the national sample. 
While both the BPS and the SCCS samples represent first-time college students, it is possible 
that there is variation in the age of first-time college students across states (perhaps reflecting 
differences in labor markets and college costs). In both the state and national data, half of Pell 
recipients are White, compared with approximately three quarters of non-Pell students. 
Approximately 70 percent of Pell recipients and 50 percent of non-Pell students are female in 
each sample. 
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Table 1 : Descriptive Statistics: National and State Samples of Community College Students 


BPS SCCS 

Variable Non-Pell Pell Non-Pell Pell 

White 


Black 

Asian 

Other race/ethnicity 

Hispanic 

Female 

Age 

State resident 

Dependent 

Work for pay 

First semester earnings ($) 

High school graduate 
Expected family contribution ($) 

N 


.735 

(.442) 

.517 

(.499) 

.090 

(.286) 

.271 

(.445) 

.044 

(.206) 

.037 

(.188) 

.105 

(.306) 

.149 

(.356) 

.135 

(.341) 

.164 

(.370) 

.526 

(.499) 

.705 

(.456) 

21.875 

(7.849) 

22.257 

(6.884) 

.954 

(.210) 

.972 

(.165) 

.668 

(.471) 

.567 

(.495) 

.786 

(.410) 

.694 

(-461) 

.879 

(.326) 

0.847 

(0.359) 

11,632.41 

(14,314.19) 

899.42 

(1,163.75) 

2,900 

1,620 


.739 

(-439) 

.516 

(.360) 

.153 

(.360) 

.380 

(.485) 

.070 

(.255) 

.079 

(.270) 

.038 

(.190) 

.025 

(.025) 

.508 

(.500) 

.673 

(.469) 

24.008 

(9.375) 

23.415 

(7.926) 

.880 

(.324) 

.953 

(.212) 

.113 

(-316) 

.519 

(.500) 

.557 

(.497) 

.478 

(.500) 

178.76 

(272.54) 

133.31 

(179.27) 

.797 

(.402) 

.884 

(.320) 

4,166.93 

(7,841.82) 

765.43 

(1,234.94) 

123,304 

50,008 


Note. Table shows averages for Pell and non-Pell community college entrants from the BPS:04/09 (N= 4,520) and 
state administrative data from fall entry cohorts from 2002-2007 (N= 173,312). We used survey weights for the 
BPS data, which represents 1,144,570 students nationally, including a subpopulation of 299,810 Pell recipients and 
844,760 non-Pell students. In correspondence with National Center for Education Statistics requirements, all sample 
sizes for the BPS are rounded to the nearest 10. 
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Financially, the national and state samples differ across dependency status, employment 
status while enrolled, and EFC. Among Pell recipients, a similar proportion filed as financial 
dependents across the two samples — 52 percent of students in the SCCS and 57 percent in the 
BPS. In the SCCS, the majority of the non-Pell students did not file for financial aid and 
therefore are considered “not dependent” for financial aid purposes (the dependency status is 
unknown and/or not applicable for two thirds of these students). Twenty-two percent fewer Pell 
recipients and 23 percent fewer non-Pell students worked for pay during the first semester in the 
SCCS than in the BPS. Finally, the EFC of non-Pell students in the national sample is much 
higher than in the SCCS. The financial differences are likely driven by the lower costs of 
attendance in the SCCS than in other public two-year systems. In the most recent cohort in our 
analytic sample (2007-2008), the maximum annual tuition and fees (16 credits per semester) at 
an SCCS college cost students approximately $1,475, about half of the national average of 
$2,708 for public two-year colleges (U.S. Department of Education, National Center for 
Education Statistics, 2013). 


4. Methods 

We use several analytic approaches to examine the magnitude of SAP failure and its 
effects. First, we demonstrate the prevalence of failure to meet SAP standards using summary 
statistics from the national and SCCS samples. To dig deeper into the effects of the policy, we 
use longitudinal SCCS administrative data and regression discontinuity (RD) and difference-in- 
differences (DD) approaches to compare students who fall above and below the SAP threshold 
for GPA. 

Failure to meet SAP standards occurs when students fail to achieve a cumulative 2.0 
GPA, to achieve a .67 credit ratio of credits attempted/credits earned, or to complete their 
program within 150 percent of the maximum stated time frame. Due to variation in maximum 
time frame across programs, we focus on the GPA and credit ratio requirements to study 
descriptive patterns. We then exploit the 2.0 GPA threshold, using cumulative GPA at the end of 
the first year of college (when students are evaluated for Pell renewal for the fall of their second 
year) to test the impacts of SAP on community college student persistence, transfer behavior, and 
degree attainment. The variation in cumulative GPA allows us to examine impacts around the 
cutoff, unlike the measure of credit ratios, which does not offer enough variation to estimate the 
effect of SAP failure. In our regression discontinuity and differences-in-differences analyses, 

SAP failure is determined by whether a student’s cumulative GPA falls below the 2.0 cutoff. 
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Regression Discontinuity 

Our RD analysis relies on the assumption that, if student characteristics are evenly 
distributed across the “treatment” threshold (the 2.0 GPA cutoff), we can attribute any 
differences in outcomes across the threshold to the policy (Imbens & Lemieux, 2008). Using the 
approach, we estimate the difference between two regression functions at the cutoff — one 
estimated below and the other estimated above. The RD estimator is the difference between the 
two regression functions at the cutoff. 

To estimate the effect of SAP, we focus on students whose cumulative GPA falls near the 
cutoff — an approach referred to as local linear regression because the estimated RD impact is 
“local” to the cutoff, relying on a narrow bandwidth of GPAs (Hahn, Todd, & Van der Klaauw, 
2001). The rationale for the local linear approach is that focusing on the subset of observations 
around the cutoff should allow us to more accurately specify the functional fonn, reducing bias in 
estimation. To identify the optimal bandwidth around the cutoff, we use Calonico, Cattaneo, and 
Titiunik’s (in press-a, in press-b) robust bias-corrected confidence intervals and Imbens and 
Kalyanaraman’s (2012) data-driven bandwidth selection. We use a bandwidth of 0.5 as our optimal 
bandwidth, based on Calonico et al.’s procedure, which results in a slightly more conservative 
estimate. In all analyses, we test for sensitivity to bandwidth selection by using the optimal 
bandwidth (0.5), half the optimal bandwidth (0.25), and twice the optimal bandwidth (1.0). 

The basic model takes the form: 

Tj = B {) + [] | ( Belowi ) + p 2 ( Distance ; * Below ;) + P 3 (. Distance ; * A hove,) + c, (1) 

where T, represents the outcome for student /', and Pi provides an estimate of the effect of falling 
below the SAP cutoff on the outcome. Below is a binary indicator of whether or not the student’s 
first-year GPA fell below the cutoff; Above indicates whether the student’s GPA fell above the 
cutoff; Distance is the difference between the student’s cumulative GPA and the 2.0 cutoff, a, is 
the residual component for student We run the basic model on each bandwidth (model 1 uses 
the optimal bandwidth; model 2, the narrow bandwidth; and model 3, the wide bandwidth). 

To increase the precision of Pi and to evaluate its sensitivity to additional covariates, we 
include institutional and cohort fixed effects, as well as individual-level measures, in additional 
models run on the optimal bandwidth. They take the fonn: 

Tj = B (} + p 1 ( Belowi ) + p 2 {Distance i * Belowi) + P 3 {Distance i * Above,) (2) 

+ CollegeFE + CohortFE + p n A + s; 

where Y, represents the outcome for student i, and Pi is the estimate of the effect of falling below 
the SAP cutoff on the outcome. CollegeFE represents a vector of institutional fixed effects 
(entered as a set of dummy variables indicating the institution initially attended, with one 
institution excluded), important because the financial aid officers responsible for implementing 
SAP policy are nested within institutions. CohortFE is a vector of cohort fixed effects, a 
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necessary inclusion because the implementation of SAP policy (including when and how 
students are provided information regarding their SAP status) may change over time. X, 
represents a vector of individual-level covariates (and [] n the corresponding estimated effects on 
the outcome) including race, gender, age at initial enrollment, state residency status, dependency 
status for the purpose of financial aid, whether the student is working for pay, earnings during 
the first semester of college, and high school graduation status. Model 4 adds the institution and 
cohort fixed effects to the basic model on the optimal bandwidth (model 1), ensuring that the RD 
is evaluated first within the institution and cohort before aggregating into Pi to calculate the main 
effect. Model 5 adds the vector of covariates to model 4 to test for sensitivity to additional 
covariates. We interpret the final model (model 5), which controls for institutional and cohort 
fixed effects, as well as student background, in the results section. 

In the context of testing the impact of SAP policy, we face an additional challenge. Given 
the nature of the state’s grading system — in which only whole letter grades are awarded — and 
the fact that many community college students do not accumulate a lot of credits (making them 
more likely to only have one or two grades comprising their cumulative GPA), we find 
“heaping” in whole number GPAs across the distribution. The first panel in Figure 1 shows the 
GPA distribution among students in our sample. To estimate discontinuities using GPA as the 
running variable, we first must deal with “heaping-induced bias” in our state administrative data 
(Barreca, Lindo, & Waddell, 2011). Following the recommendations of Barreca, Lindo, and 
Waddell (2011; see also Barreca, Guldi, Lindo, & Waddell, 2011), we rely on “donut-RD” 
estimates, dropping observations at the whole number heaps that fall within our bandwidth." 
While “lumpy” distributions for the running variable are often taken as evidence that there is 
manipulation of the running variable (McCrary, 2008), we have no reason to expect that the 
heaping is induced by student manipulation of grades. While feasible, it is highly unlikely that 
students, especially since many are unaware of the SAP standards, are manipulating their course 
grades to fall above the cutoff. However, we agree that including the “whole” GPAs (i.e., 0.0, 

1.0, 2.0, 3.0, and 4.0) introduces bias; students with whole GPAs may be less likely to persist 
because the presence of a whole GPA is often the result of fewer course credits contributing to 
the calculated average. The second panel of Figure 1 shows the distribution across the wide 
bandwidth after removing the heaps. 


2 A drawback of this method is that it cannot estimate how SAP impacts students who tend to be observed in the 
heaps (in this case, the heaps appear to include students with few credits accumulated). Still, the method is 
consistent with the usual motivation for RD and, based on simulations in Barreca, Lindo, and Waddell (2011), offers 
unbiased estimates. 
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Figure 1 : Changes in GPA Distribution Among Pell Grant Recipients When Eliminating Heaping 
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Note. The first panel shows the distribution of student GPAs across the wide bandwidth (within 1 grade point of the 
cutoff), with heaping in the bin directly above the 2.0 cutoff. The second panel shows the distribution of student 
GPAs used for donut-RD analysis (in which students with “whole” GPAs are removed from analytic sample). 

Source: SCCS administrative data. 


Difference-in-Differences 

According to federal legislation, an institution’s SAP policy must be at least as strict as 
the academic requirements for graduation. In the SCCS, the stated academic policy for 
graduation and for remaining in good academic standing is a 2.0 cumulative GPA. This means 
that our RD estimates combine any general effects of failing institutional standards for academic 
performance with the specific effects of failing SAP standards for retaining financial aid. Thus, 
we also apply a difference-in-differences (DD) approach to compare the effects of falling below 
the GPA cutoff among Pell Grant recipients (who face both SAP and general institutional 
consequences) and non-Pell students (who face only general institutional consequences). In 
essence, we use non-Pell students as a control group for a comparison of the effects of falling 
below the SAP GPA standard. 

The logic of DD estimation is based on the idea that outcomes are observed for two 
different subgroups. One of the subgroups, Pell recipients, was exposed to a policy, SAP, and 
some of the students within this group fell below the threshold, presumably experiencing the 
threat of Pell loss. The other subgroup is comprised of students who did not receive the Pell at 
college entrance and were not exposed to the SAP policy. When GPA values are observed in a 
setting of SAP exposure versus non-exposure, the difference in gains (or losses) in the 
unexposed non-Pell students who fell below a cumulative GPA of 2.0 is subtracted from the 
gains (or losses) of the SAP-exposed Pell students (those who failed to meet SAP). This 
approach removes biases of exposure to general academic regulations (which encourage students 
to stay above the 2.0 GPA mark in order to graduate). 
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We compare Pell and non-Pell students falling below and above the cutoff using the 
following basic model (model 1 ) on the optimal bandwidth: 

Y, = B {] + [ : ! | (Pell, * Belowi ) + (3 2 ( Pelk ) + (3 3 ( Belowi ) + Si (3) 

where 7; represents the outcome for student i, Pell is a binary indicator of whether the student 
received the Pell Grant at initial enrollment, and Below is a binary indicator of whether or not the 
student’s GPA fell below the cutoff. Coefficient pi represents the effect of falling below the 2.0 
GPA threshold for students who entered college with a Pell versus those who did not (the 
difference-in-differences between students who fall below and above the 2.0 GPA threshold at 
each Pell condition). P 2 is the effect of Pell on the outcome; P 3 is the effect of falling below the 
cutoff on the outcome. £, is the residual component for student i. 

To increase the precision of pi and to evaluate its sensitivity to additional covariates, we 
conduct a series of stepwise regression analyses, adding institutional and cohort fixed effects and 
individual-level covariates to the basic model: 

7 = B {) + P 1 (Pell, * Belowi ) + P 2 ( Pell y ) + p 3 ( Below\ ) + CollegeFE (4) 

+ CohortFE + \i n X, + si 

In this model, Y\ represents the outcome, Pell is a binary indicator of whether the student 
received the Pell Grant at initial enrollment, Below is a binary indicator of whether or not the 
student’s GPA fell below the cutoff, CollegeFE represents a vector of institutional fixed effects, 
and CohortFE represents a vector of cohort fixed effects. Coefficient pi represents the effect of 
falling below the 2.0 GPA threshold for students who entered college with a Pell versus those 
who did not in a given institution and cohort. X, represents a vector of individual-level covariates 
(described below) added in a series of stepwise regressions. Sj is the residual component for 
student i. 

In model 2, we add in institutional and cohort fixed effects to control for variation across 
institutions and students who enter college in different years, but we do not yet include student- 
level covariates. Model 3 adds the distance of students’ GPA from the cutoff ( Distance i) to 
model 2. In order to capture the variation in slopes across Pell status observed above and below 
the GPA threshold (see Figure 4B), model 4 adds the student’s GPA relative to the cutoff, 
Distance \ * Belowi, to the previous model. Our most comprehensive model, model 5, adds 
measures of race, gender, age at initial enrollment, state residency status, dependency status for 
the purpose of financial aid, whether the student is working for pay, earnings during the first 
semester of college, and high school graduation status. We interpret the results from model 5 in 
the results section. 
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5. Results 


How Many Students Are Potentially Affected by SAP Requirements? 

We begin by isolating the GPA component of the SAP requirements, which is more 
uniform across institutions and also easier to construct in the national and state data. As we 
noted, a cumulative GPA of 2.0 is a commonly accepted GPA criterion of SAP, including in our 
institutional sample (where it is assessed at the end of each tenn, rather than at the end of each 
year). 

National trends. Table 2 shows the percentage of first-year Pell recipients (i.e., Pell 
recipients with first-year academic standing) who had a cumulative GPA of less than 2.0 at the 
end of the year. Twenty-one percent of first-year Pell recipients in 2012 failed to meet the SAP 
GPA standard. The rate rose over time, particularly at public two-year colleges. While rates of 
first-year GPA failure in 2012 were highest at community colleges, the problem is far from 
unique to these institutions. First-year GPA failure rates at public four-year institutions were 
barely lower in 2012, and in prior years were higher than at two-year institutions. 

Table 2 also shows that the issue is not unique to Pell recipients. While Pell recipients 
have somewhat higher rates of GPA failure than other students — not necessarily surprising, 
given their socioeconomic risk factors — the differences are not dramatic. In 2012, 19 percent of 
non-Pell first-year students would have failed to meet the GPA standard for SAP, including 2 1 
percent of first-year students at community colleges. 


Table 2: Recent National Trends in SAP GPA Failure 




Pell (%) 



Non-Pell (%) 


Institution Sector 

NPSAS:04 

NPSAS:08 

NPSAS:12 

NPSAS:04 

NPSAS:08 

NPSAS:12 

Public four-year 

22.6 

24.3 

24.2 

17.3 

19.0 

18.7 

Private not-for-profit 
four-year 

19.7 

16.0 

15.3 

9.3 

11.0 

8.0 

Public two-year 

17.0 

20.4 

24.5 

14.8 

18.3 

20.5 

Private for-profit 

13.6 

16.1 

15.1 

8.6 

12.8 

12.1 

Attended more than 
one college 

14.8 

15.8 

18.0 

12.0 

12.4 

13.5 

Total 

17.3 

18.9 

21.0 

15.1 

17.5 

19.2 


Note. Table displays percentage of students failing to achieve a 2.0 or higher GPA in the given academic year, 
estimated using National Postsecondary Student Aid Study 2004, 2008, and 2012 data on first-year-equivalent 
students. 
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Community college administrative data. We now move to studying patterns in our 
administrative data, which allow for a closer look at academic perfonnance and subsequent 
outcomes. In these data, we examine the credit requirement as well as the GPA requirement for 
SAP. We are able to look at these measures by academic term, as they are evaluated in the 
SCCS, rather than by academic year. Figure 2 plots rates of failure for the GPA requirement, the 
credit requirement, and SAP overall between 2002 and 2008. 

The figure indicates increasing rates of failure over time for the GPA standard, consistent 
with the national data for community colleges. For the years in which we have comparable data, 
the levels of failure for the GPA requirement are slightly higher in the SCCS sample. This may 
simply reflect regional variation, or it may be due to the fact that our administrative data 
represent true first-time entrants in their first term, while the NPSAS data span an entire 
academic year (so more data points are included in the average) and include all students with 
first-year academic standing. 

Figure 2 also indicates that at least as many students are failing the credit standard as are 
failing the GPA standard. Of course, there is significant overlap between the two requirements, but 
accounting for students who fail either of the two standards raises the overall first-term SAP failure 
rate substantially, from 25 percent to 42 percent for students who entered in the fall of 2008. 


Figure 2: First-Term SAP Failure Rates by Entry Cohort (All Beginning First-Years) 



Note. N= 173,312. The figure presents the proportion of first-time SCCS students who fail to meet the GPA, credit, 
and, consequently, overall SAP standards within their first term of enrollment. Due to problems with the credits 
attempted measure in 2005 and 2006, we were unable to calculate students’ credit ratio, and therefore the overall 
SAP failure estimate, for 2005-2006. 

Source: SCCS administrative data. 
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What Happens to Students Who Fail SAP Over Time? 

Descriptive results. To the extent that SAP failure has a consequence, it is likely show 
up as dropout rather than as persistence without financial aid. Figure 3 shows that SAP failure is 
most prevalent in the first term of enrollment. Approximately 40 percent of Pell recipients in the 
SCCS failed SAP in the first term. By the fall of the second year, the rate is cut in half (to 18 
percent), and by the fall of the third year, the rate is halved again (to 9 percent of Pell entrants). 
The reduction in SAP failure appears to be a function of dropout. Figure 3 shows that the 
prevalence of SAP failure declines over time not because students improve their GPAs overall 
(though, of course, some do) but because many students simply do not reenroll, and those who 
fail SAP are disproportionately likely to drop out (29 percent of Pell recipients who fail SAP in 
the first term do not return in the spring, compared with just 13 percent of Pell recipients who 
meet SAP standards). The students who return in subsequent semesters are thus positively 
selected and less at risk of SAP failure to begin with. 


Figure 3: Pell Entrants' Enrollment by SAP Status Across Academic Terms 



Fall, Y1 Spring, Y1 Fall, Y2 Spring, Y2 Fall, Y3 Spring, Y3 

Term 

■ Enrolled, Failing SAP ■ Enrolled, Passing SAP Not Enrolled 


Note. N = 50,008. The figure presents Pell entrants’ enrollment and SAP status over time (excluding summer terms, 
when fewer students are enrolled). In the first term, a small percentage of enrolled students had no valid GPA. 
Otherwise, these categories are mutually exclusive and should add to 100 percent. 

Source: SCCS administrative data. 
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We find that, among SCCS students who remained enrolled after failing SAP, the 
majority continued to receive Pell Grants, likely as a result of successful appeals. In the fall term 
of students’ second year, 92 percent of students who failed to meet SAP standards and were still 
enrolled continued to receive the Pell Grant. This high rate of Pell receipt continued to occur 
even after the stated probation period ended (students have one semester to improve their SAP 
status while on probation-by-appeal). Among students who continued to fail to meet SAP 
standards and reenrolled in the fall term of their third year, 94 percent received the Pell. While 
this may sound alarming, these students represent a very small percentage of the population — 6 
percent — because only about 7 percent of students who continued to fail to meet SAP standards 
remained enrolled for that long. 

While students who fail to meet SAP requirements are more likely to drop out, the 
difference in retention rates between those who do and do not fail SAP has no obvious causal 
interpretation. The preexisting factors that lead to low GPAs likely also lead to low persistence 
rates. Moreover, since SAP rules for Pell typically correspond to institutions’ broader academic 
progress requirements, it is difficult to separately identify the Pell-related consequences of SAP 
failure. 

There are several ways to parse out the effects of SAP failure. One way to approach this 
issue is to compare the Pell recipients directly above and below the cutoff using a regression 
discontinuity design. Another is to look at persistence rates by GPA for those with and without 
Pell Grants, to allow for a difference-in-differences analysis. Figure 4 provides preliminary 
evidence for both of these methods, plotting GPA and persistence over time for Pell and non-Pell 
students. Panel A looks at fall-to-spring persistence by first-term GPA, while panel B looks at 
fall-to-fall persistence by cumulative first-year GPA (those who drop out during the first year 
remain in the sample, and have their GPA at exit carried forward). 

Interestingly, panel A indicates that Pell recipients persist to the spring term at higher 
rates than non-Pell students across the GPA spectrum. We do not see any strong changes in 
patterns of persistence around the 2.0 GPA cutoff, nor do we particularly expect to see any, 
given that there are no distinct consequences for falling below this threshold until at least the 
second year (unless there are discouragement effects from receiving a warning status). Panel B 
looks quite different. Here, students below the 2.0 GPA threshold who entered with Pell Grants 
appear to have lower rates of persistence to the subsequent fall term than non-Pell entrants with 
similar GPAs; above the threshold, the two groups persist at similar rates. It is plausible that the 
SAP rules embedded in the Pell Grant program help explain these patterns: The threat of losing 
the Pell due to SAP rules may discourage students who entered with the Pell from continuing to 
the fall of their second year. We dig deeper into these patterns in our subsequent analyses. 
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Figure 4. Early College Persistence Patterns for Pell and Non-Pell Students 


A. Fall-to-Spring Persistence, 
by First-Term GPA and Pell Status 
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First-Term GPA 


Pell 

— No Pell 


B. Fall-to-Fall Persistence, 
by First-Year GPA and Pell Status 
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First-Term GPA 
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No Pell 


Note. N= 173,312. In both panels, data for whole GPAs (1.00, 2.00, 3.00, and 4.00) are hidden. 
Source: SCCS administrative data. 
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Regression discontinuity. The RD analysis is our first effort to more closely examine the 
potential discouragement effect of the SAP standards. To bolster support for an RD approach 
using students’ first-year GPA as the running variable, we first test for discontinuities in 
covariates around the 2.0 cutoff. As demonstrated in Table 3, regressions confirm that there are 
no systematic differences between Pell recipients who fell directly below and above the cutoff in 
tenns of race, age, whether the student was working for pay, whether the student graduated from 
high school, and several other observable measures. We find some evidence that women were 
more likely to fall below the cutoff than men. We include gender in our covariate-adjusted 
models to control for this discontinuity. 


Table 3: Regression Discontinuity Estimates for Covariates 


Outcomes 

Optimal Bandwidth 

Narrow Bandwidth 

Wide Bandwidth 

White 

0.010 

-0.009 

0.022 


(0.022) 

(0.036) 

(0.024) 

Black 

-0.009 

0.008 

-0.015 


(0.022) 

(0.036) 

(0.024) 

Other race 

0.004 

0.013 

-0.002 


(0.013) 

(0.021) 

(0.014) 

Hispanic 

-0.005 

-0.012 

-0.005 


(0.007) 

(0.011) 

(0.008) 

Female 

0.053* 

0.020 

0.052* 


(0.021) 

(0.035) 

(0.023) 

Age 

-0.269 

-0.282 

0.003 


(0.285) 

(0.460) 

(0.340) 

State resident 

0.008 

0.010 

0.009 


(0.009) 

(0.014) 

(0.010) 

Dependency status 

-0.013 

-0.001 

-0.006 


(0.025) 

(0.040) 

(0.028) 

Working for pay 

0.014 

0.002 

0.025 


(0.022) 

(0.036) 

(0.024) 

First semester earnings 

2.881 

2.401 

9.705 


(6.722) 

(11.021) 

(7.878) 

High school graduate 

0.010 

0.012 

0.008 


(0.014) 

(0.012) 

(0.016) 

N 

10,578 

4,445 

21,884 


Note. Table presents RD estimator coefficients, with standard errors in parentheses. The functional form for the 
optimal (0.5) and narrow (0.25) bandwidths is local linear, while the functional form for the wide (1.0) bandwidth is 
local quadratic. 

*p < .05; **p < .01; ***p < .001. 

Source: SCCS administrative data. 
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Next, we run our main RD analyses. Table 4 shows the effect of failing to meet the GPA 
standard for SAP at the end of the first year of college, when students in the warning period 
should face the threat of Pell loss, on early college persistence (reenrollment in the second year of 
college), first-term GPA from the second-year, certificate and associate degree attainment, and 
transfer to a four-year college. While we might expect to see an immediate negative impact on 
persistence (whether from Pell loss or discouragement due to the threat of Pell loss), the RD 
results do not support that hypothesis. Our results are imprecise, and small negative effects cannot 
be ruled out. However, our point estimates show that Pell recipients who fell immediately below 
the 2.0 GPA threshold were slightly more likely to reenroll in the second year than those who fell 
immediately above (fl = .038, SE = 0.023 in model 5, our preferred model). This difference is not 
statistically significant. GPA failure had a small negative effect on longer tenn outcomes, such as 
associate degree attainment and transfer to a four-year college, but again, the differences are not 
statistically significant. To provide additional insight into the distribution of student outcomes 
across GPA, Appendix A includes figures that present the RD results graphically. 


Table 4: Regression Discontinuity Effects of SAP on College Outcomes Within 3 Years of Entry 


Outcomes 

Model 1 

Model 2 

Model 3 

Model 4 

Model 5 

Persistence into year 2 

0.035 

0.019 

0.003 

0.038 

0.038 


(0.023) 

(0.044) 

(0.027) 

(0.023) 

(0.023) 

First-term, second-year GPA 

-0.006 

0.025 

-0.011 

-0.006 

-0.007 


(0.036) 

(0.067) 

(0.040) 

(0.036) 

(0.035) 

Certificate 

-0.002 

-0.012 

-0.006 

-0.001 

-0.001 


(0.005) 

(0.009) 

(0.006) 

(0.005) 

(0.005) 

Associate degree 

-0.008 

-0.011 

-0.015* 

-0.007 

-0.007 


(0.006) 

(0.012) 

(0.007) 

(0.006) 

(0.006) 

Transfer to four-year college 

-0.029 

-0.027 

-0.04 

-0.037 

-0.035 


(0.019) 

(0.036) 

(0.022) 

(0.019) 

(0.019) 

N 

10,578 

4,445 

21,884 

10,578 

10,578 



Specifications 




Bandwidth 

0.5 

0.25 

1.0 

0.5 

0.5 

Form 

Linear 

Linear 

Quadratic 

Linear 

Linear 

Fixed effects 

No 

No 

No 

Yes 

Yes 

Covariates 

No 

No 

No 

No 

Yes 


Note. Table presents RD estimator coefficients, with standard errors in parentheses. The functional form for the 
optimal (0.5) and narrow (0.25) bandwidths is local linear, while the functional form for the wide (1.0) bandwidth is 
local quadratic. “Fixed effects” refer to institutional and cohort fixed effects, included in the models as a series of 
dummy variables with the reference eliminated. To put results into context, the sample average for each outcome: 
persistence = 0.518, second-year GPA = 2.449, certificate = 0.023, associate degree = 0.074, transfer = 0.255. 

*p < .05; **p < .01; ***p < .001. 

Source: SCCS administrative data. 
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Difference-in-differences. The DD results, presented in Table 5 (see Appendix B for a 
graphical presentation of results), expand on the patterns observed in Figure 4, panel B. In Figure 
4, panel B, differences in persistence among Pell and non-Pell students appear to be driven by 
differences below the GPA cutoff, as the Pell students appear to reach parity with non-Pell 
students a little above the 2.0 mark. The DD results, in contrast with the RD results (which focus 
on Pell recipients only), suggest a negative effect of SAP policy on Pell recipients by comparing 
the differences in outcomes above and below the threshold between Pell and non-Pell students. 
The DD results are also more precise. Pell students who fail to meet the GPA standard for SAP 
are 4 percentage points less likely to persist to the second year of college than non-Pell students 
who fail to earn a 2.0 GPA but do not face SAP standards ((1 = -0.038, SE = 0.012 in our 
preferred model, model 5). 


Table 5: Difference-in-Differences Results for SAP Impact on College Outcomes 


Outcomes 

Model 1 

Model 2 

Model 3 

Model 4 

Model 5 

Persistence into year 2 

-0.040*** 

(0.012) 

-0.037** 

(0.012) 

-0.039*** 

(0.012) 

-0.039** 

(0.012) 

-0.038** 

(0.012) 

First-term, second-year GPA 

0.020 

(0.018) 

0.022 

(0.018) 

0.015 

(0.018) 

0.015 

(0.018) 

0.018 

(0.018) 

Certificate 

0.001 

(0.003) 

0.001 

(0.003) 

0.000 

(0.003) 

0.000 

(0.003) 

0.001 

(0.003) 

Associate degree 

0.014*** 

(0.004) 

0.014*** 

(0.004) 

0.013*** 

(0.005) 

0.013*** 

(0.004) 

0.013*** 

(0.004) 

Transfer to four-year college 

0.029** 

(0.010) 

0.032** 

(0.010) 

0.031** 

(0.010) 

0.031** 

(0.010) 

0.030** 

(0.010) 



Specifications 




Fixed effects 
Covariates 

No 

Yes 

Yes 

Yes 

Yes 

Distance 

No 

No 

Yes 

Yes 

Yes 

Distance*Below 

No 

No 

No 

Yes 

Yes 

Controls 

No 

No 

No 

No 

Yes 


Note. N= 31,768. Table presents DD estimator coefficients, with standard errors in parentheses. Each model is 
performed on the optimal (0.5) bandwidth with a linear functional form. “Fixed effects” refer to institutional and 
cohort fixed effects, included in the models as a series of dummy variables with the reference eliminated. 

*p < .05; **p < .01; ***p < .001. 

Source: SCCS administrative data. 


24 


While the SAP policy appears to negatively impact early college persistence, it may 
improve degree attainment and transfer to a four-year college among students who persist. We 
find a small positive effect on associate degree attainment and transfer to a four-year college 
within three years of college entrance. Pell recipients who failed to meet SAP standards in their 
first year were approximately 1 percentage point more likely to earn an associate degree and 3 
percentage points more likely to transfer to a four-year college than their non-Pell peers who also 
fell below the 2.0 mark in their first year. Anticipating that the small positive impacts on 
associate degree attainment and transfer may indicate an incentivizing effect on student effort, 
we estimate the impact of the SAP policy’s GPA standards on first-term, second-year GPA. The 
effect is positive but small and not statistically significant. Pell students who return for a second 
year of college after failing to meet the GPA standard in the first year perform almost 2 
percentage points better, in terms of first-term GPA, than their non-Pell peers who also fell 
below a 2.0 GPA in their first year of college. 


6. Discussion 

In this paper, we provide preliminary evidence regarding the prevalence and 
consequences of students’ failure to meet SAP standards, using both national data and state 
administrative data from community colleges. The trends highlighted in the data suggest that 
failure to meet academic standards is not a Pell recipient-focused phenomenon, nor is it 
concentrated in community colleges. While many students, not just Pell recipients, fail to meet 
minimum academic standards, we expect that the consequences of failing to meet academic 
standards are greatest for Pell recipients because they may lose their eligibility to retain financial 
aid. Failure to meet academic standards may be especially problematic at community colleges, 
where many students from low-income families and with lower academic preparation are 
concentrated. 

A substantial portion of Pell recipients fail to meet SAP requirements. Forty percent of 
SCCS students failed to meet SAP requirements in 2012. If the policy has teeth and ultimately 
results in Pell ineligibility, it could affect a large portion of low-income students. SAP policy 
poses an additional hurdle for students who have financial constraints. Even if there are some 
incentivizing effects, the discouragement effects raise the question of how punitive policies fit 
into an equity agenda for American higher education. Gaining greater insight into how SAP 
policy sorts students is necessary to improve policy implementation. 

Our descriptive evidence and the results from our DD analysis offer some evidence in 
support of a discouragement effect. Pell recipients (who were subject to both SAP and 
institutional academic progress standards) experienced greater negative effects of falling below 
the 2.0 GPA threshold than their non-Pell peers (who were subject to only institutional general 
academic standards). At the same time, we cannot entirely rule out the theory that SAP policy 
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may incentivize effort among some students, which may explain the higher rates of associate 
degree attainment and transfer to a four-year college among Pell students who failed to meet 
SAP standards but persisted, compared with students who were not receiving the Pell. 

The small positive effect of failure to meet the SAP GPA standard on early persistence 
obtained from the RD analysis is imprecise and not statistically significant, and thus we caution 
against overinterpretation. However, it does leave open the possibility that, among Pell 
recipients, those who just barely fail to meet SAP standards are more likely to reenroll that those 
who just barely meet the requirements. This may be explained by a “second-chance” 
phenomenon, where students who fall below the cutoff are made aware that, while their grades 
are poor, they are being given a second chance and may even be targeted for additional 
institutional supports. Those directly above the cutoff, who are not perfonning substantially 
better, do not receive this signal of federal or institutional support and are left to interpret their 
academic and financial status on their own. 

Another way to resolve the conflicting estimates of SAP failure’s effect on early college 
persistence in the RD and DD analyses is to consider the goals of the two analytic approaches. 
The RD analysis is limited to Pell recipients, and the analysis is local to the cutoff, comparing 
Pell students who just missed meeting SAP requirements to Pell students who just barely met 
them. The RD analysis also captures the effect of academic probation in general, while the DD 
analysis isolates the effect of federal SAP failure above and beyond academic probation. When 
considering the effect on Pell recipients without a comparison group, as we do in the RD 
analysis, we find little evidence of significant effects of SAP standards on college outcomes. 
When comparing the effects of falling below a 2.0 GPA between Pell recipients and their non- 
Pell peers, as in the DD analysis, we see that SAP standards appear to reduce early college 
persistence. 

We attempt to further evaluate whether SAP incentivizes effort by examining term- 
specific GPA immediately after students would receive notification of SAP failure and file for an 
appeal (in the first term of the second year). The small, positive DD estimates of SAP failure on 
the subsequent GPA of returning students are not statistically significant. Still, the direction of 
the GPA effects, combined with the positive DD impacts of SAP on degree attainment and 
transfer, leave the debate open. It is possible that, as Lindo and his colleagues (2010) find, there 
is heterogeneity by prior academic performance, and that some students are particularly 
susceptible to discouragement effects (in the case of academic probation, the authors found that 
students with previously high achievement were more likely to drop out if they failed to meet 
academic standards). Unfortunately, the SCCS data do not include any measure of prior 
achievement. 

Data limitations preclude some additional analyses that also would be of interest. For 
example, while the SCCS administrative data provide us with measures of students’ grades, 
credits, and outcomes, we do not have access to their SAP status, as determined by their financial 
aid office. It is possible that many students facing Pell loss due to SAP failure avoid negative 
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consequences by appealing. While federal and institutional policy suggest that students cannot 
delay negative effects for too long — an appeal technically only provides one additional semester 
to meet SAP standards, and all of the standards are cumulative — many students who remain 
enrolled despite repeated failure to meet SAP standards retain the Pell. Additionally, as we 
suggest above, additional infonnation about students’ prior achievement could help us pinpoint 
which students are most at risk of SAP failure and negative effects, but we do not have access to 
measures of precollege academic performance. 

Overall, we find the DD results most compelling, both because they are more precise and 
because they isolate the effect of federal SAP failure. These results suggest that failing to meet 
SAP standards may negatively impact persistence but positively impact eventual transfer and 
associate degree attainment — suggesting heterogeneous effects across students. Nonetheless, 
drawing strong conclusions from these single-state estimates would be premature. While the 
analytic approaches are sound for the study’s goals, the results are not always consistent with 
hypotheses, and the contrast between the RD and DD results suggests that more research is 
necessary to confirm the patterns observed here. This study serves as a point of departure from 
which policy scholars may continue exploring the implications of tying need-based aid to 
performance standards. 

To implement the federal SAP policy, institutions invest resources into evaluating SAP. 
In the SCCS, the information gleaned from this evaluation is used to notify students of their SAP 
status. But the assessment of student progress could be better utilized to improve college 
completion among students from low-income backgrounds. Considering the magnitude of SAP 
failure and the resources put into its assessment, using the data to implement an early warning 
system — notifying students early in their college career of their academic status and of resources 
available to improve it — would be practical and potentially more effective than merely notifying 
students of their status (Goldrick-Rab, 2013b). Unfortunately, at most campuses, different 
student service offices and their staff are siloed. Even if financial aid officers are aware that 
students risk losing their financial aid, the students may never be contacted for academic 
intervention. An early warning system to notify students of SAP failure, advise them on 
appropriate steps for improving their academic standing, and link them with support services 
may be a worthwhile investment for colleges interested in improving the outcomes of low- 
income students. While we cannot speak to the effectiveness of more tightly coupling financial 
aid to academic advising and other student services, we hope that our examination of SAP 
policies will encourage more research in this area. 
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Appendix A: Regression Discontinuity Results 


Figure A.1 : Impact of SAP on Persistence Into Second Year 



Note. The graph presents scatterplots of average outcomes within 0.05-point bins across the wide bandwidth (1.0 
grade points on each side of the cutoff) and fitted lines representing the average effects for Pell recipients on each 
side of the cutoff across the optimal bandwidth (0.5 grade points on each side). 

Source: SCCS administrative data. 
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Figure A.2: Impact of SAP on First-Term, Second-Year GPA 



Note. The graph presents scatterplots of average outcomes within 0.05-point bins across the wide bandwidth (1.0 
grade points on each side of the cutoff) and fitted lines representing the average effects for Pell recipients on each 
side of the cutoff across the optimal bandwidth (0.5 grade points on each side). 

Source: SCCS administrative data. 
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Figure A.3: Impact of SAP on Certificate Attainment Within 3 Years 



Note. The graph presents scatterplots of average outcomes within 0.05-point bins across the wide bandwidth (1.0 
grade points on each side of the cutoff) and fitted lines representing the average effects for Pell recipients on each 
side of the cutoff across the optimal bandwidth (0.5 grade points on each side). 

Source: SCCS administrative data. 
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Figure A.4: Impact of SAP on Associate Degree Attainment Within 3 Years 



Note. The graph presents scatterplots of average outcomes within 0.05-point bins across the wide bandwidth (1.0 
grade points on each side of the cutoff) and fitted lines representing the average effects for Pell recipients on each 
side of the cutoff across the optimal bandwidth (0.5 grade points on each side). 

Source: SCCS administrative data. 


36 


Figure A.5: Impact of SAP on T ransfer to a Four-Year College Within 3 Years 



Note. The graph presents scatterplots of average outcomes within 0.05-point bins across the wide bandwidth (1.0 
grade points on each side of the cutoff) and fitted lines representing the average effects for Pell recipients on each 
side of the cutoff across the optimal bandwidth (0.5 grade points on each side). 

Source: SCCS administrative data. 
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Appendix B: Difference-in-Differences Results 


Figure B.1 : Impact of SAP on Persistence Into the Second Year of College 
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Note. The graphs present scatterplots of the average outcomes for Pell and non-Pell students in 0.05 point-bins 
across the wide bandwidth (1.0 grade points on each side of the cutoff) and fitted lines representing the average 
effects for Pell recipients (solid line) and non-Pell students (dotted line) across the optimal bandwidth (0.5 grade 
points on each side of the cutoff). 

Source: SCCS administrative data. 
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Figure B.2: Impact of SAP on First-Term, Second-Year GPA 
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Note. The graphs present scatterplots of the average outcomes for Pell and non-Pell students in 0.05 point-bins 
across the wide bandwidth (1.0 grade points on each side of the cutoff) and fitted lines representing the average 
effects for Pell recipients (solid line) and non-Pell students (dotted line) across the optimal bandwidth (0.5 grade 
points on each side of the cutoff). 

Source: SCCS administrative data. 
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Figure B.3: Impact of SAP on Certificate Attainment Within 3 Years 
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Note. The graphs present scatterplots of the average outcomes for Pell and non-Pell students in 0.05 point-bins 
across the wide bandwidth (1.0 grade points on each side of the cutoff) and fitted lines representing the average 
effects for Pell recipients (solid line) and non-Pell students (dotted line) across the optimal bandwidth (0.5 grade 
points on each side of the cutoff). 

Source: SCCS administrative data. 


40 


Figure B.4: Impact of SAP on Associate Degree Attainment Within 3 Years 
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Note. The graphs present scatterplots of the average outcomes for Pell and non-Pell students in 0.05 point-bins 
across the wide bandwidth (1.0 grade points on each side of the cutoff) and fitted lines representing the average 
effects for Pell recipients (solid line) and non-Pell students (dotted line) across the optimal bandwidth (0.5 grade 
points on each side of the cutoff). 

Source: SCCS administrative data. 
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Figure B.5: Impact of SAP on Transfer to a Four-Year College Within 3 Years 
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Note. The graphs present scatterplots of the average outcomes for Pell and non-Pell students in 0.05 point-bins 
across the wide bandwidth (1.0 grade points on each side of the cutoff) and fitted lines representing the average 
effects for Pell recipients (solid line) and non-Pell students (dotted line) across the optimal bandwidth (0.5 grade 
points on each side of the cutoff). 

Source: SCCS administrative data. 
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